Mosaic oblique images and systems and methods of making and using same

ABSTRACT

Systems and methods for creating oblique-mosaic image(s) for geographical area(s) are disclosed, including a computer system running software that when executed causes the system to create a mathematical model of a sensor of a virtual camera having an elevation greater than an elevation of a desired geographical area to be imaged, the mathematical model having an oblique-mosaic pixel map; determine surface locations for pixels included in the oblique-mosaic pixel map; select source oblique images of the surface locations of the pixels captured at an oblique angle and compass direction similar to an oblique angle and compass direction of the virtual camera; and reproject source oblique image pixels for pixels included in the oblique-mosaic pixel map such that reprojected pixels have differing sizes determined by matching projections from the virtual camera so as to present an oblique appearance, and thereby create the oblique-mosaic image of the desired geographical area.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present patent application claims priority to and is a continuationof U.S. Ser. No. 15,965,222, filed Apr. 27, 2018, which is acontinuation of U.S. Ser. No. 15/797,679, filed Oct. 30, 2017, now U.S.Pat. No. 9,959,653, which is a continuation of U.S. Ser. No. 15/253,135,filed Aug. 31, 2016, now U.S. Pat. No. 9,805,489, which is acontinuation of U.S. Ser. No. 14/152,366, filed Jan. 10, 2014, now U.S.Pat. No. 9,437,029, which is a continuation of U.S. Ser. No. 13/778,708,filed Feb. 27, 2013, now U.S. Pat. No. 8,660,382, which is acontinuation of U.S. Ser. No. 13/317,499, filed Oct. 19, 2011, now U.S.Pat. No. 8,452,125, which is a continuation of U.S. Ser. No. 12/930,546,filed Jan. 10, 2011, now U.S. Pat. No. 8,081,841, which is acontinuation of U.S. Ser. No. 11/847,125, filed Aug. 29, 2007, now U.S.Pat. No. 7,873,238, which claims priority to the provisional patentapplication identified by U.S. Ser. No. 60/840,992 filed on Aug. 30,2006, the entire contents of all of which are hereby incorporated hereinby reference in their entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The presently claimed and disclosed invention(s) relate to mosaickedoblique images and methods for making and using same. More particularly,the presently claimed and disclosed invention(s) use a methodologywhereby separate obliquely captured aerial images are combined into atleast one single oblique mosaic image. The at least one singleoblique-mosaic image is visually pleasing and geographically accurate.

2. Background of the Art

In the remote sensing/aerial imaging industry, imagery is used tocapture views of a geographic area and be able to measure objects andstructures within the images as well as to be able to determinegeographic locations of points within the image. These are generallyreferred to as “geo-referenced images” and come in two basic categories:

-   -   1. Captured Imagery—these images have the appearance they were        captured by the camera or sensor employed.    -   2. Projected Imagery—these images have been processed and        converted such that they conform to a mathematical projection.

All imagery starts as captured imagery, but as most software cannotgeo-reference captured imagery, that imagery is then reprocessed tocreate the projected imagery. The most common form of projected imageryis the ortho-rectified image. This process aligns the image to anorthogonal or rectilinear grid (composed of rectangles). The input imageused to create an ortho-rectified image is a nadir image—that is, animage captured with the camera pointing straight down.

It is often quite desirable to combine multiple images into a largercomposite image such that the image covers a larger geographic area onthe ground. The most common form of this composite image is the“ortho-mosaic image” which is an image created from a series ofoverlapping or adjacent nadir images that are mathematically combinedinto a single ortho-rectified image.

Each input nadir image, as well as the output ortho-mosaic image, iscomposed of discrete pixels (individual picture elements) of informationor data. As part of the process for creating an ortho-rectified image,and hence an ortho-mosaic image, an attempt is made to reproject (movewithin a mathematical model) each pixel within the image such that theresulting image appears as if every pixel in the image were a nadirpixel —that is, that the camera is directly above each pixel in theimage.

The reason this ortho-rectification process is needed is it is notcurrently possible to capture an image where every pixel is nadir to(directly below) the camera unless: (1) the camera used is as large asthe area of capture, or (2) the camera is placed at an infinite distanceabove the area of capture such that the angle from the camera to thepixel is so close to straight down that it can be considered nadir. Theortho-rectification process creates an image that approximates theappearance of being captured with a camera where the area on the groundeach pixel captures is considered nadir to that pixel, i.e. directlybelow that pixel. This process is done by creating a mathematical modelof the ground, generally in a rectilinear grid (a grid formed ofrectangles), and reprojecting from the individual captured camera imageinto this rectilinear grid. This process moves the pixels from theirrelative non-nadir location within the individual images to their nadirpositions within the rectilinear grid, i.e. the image is warped to lineup with the grid.

When creating an ortho-mosaic, this same ortho-rectification process isused, however, instead of using only a single input nadir image, acollection of overlapping or adjacent nadir images are used and they arecombined to form a single composite ortho-rectified image known as anortho-mosaic. In general, the ortho-mosaic process entails the followingsteps:

-   -   A rectilinear grid is created, which results in an ortho-mosaic        image where every grid pixel covers the same amount of area on        the ground.    -   The location of each grid pixel is determined from the        mathematical definition of the grid. Generally, this means the        grid is given an X and Y starting or origin location and an X        and Y size for the grid pixels. Thus, the location of any pixel        is simply the origin location plus the number of pixels times        the size of each pixel. In mathematical terms:        X_(pixel)=X_(origin)+X_(size)×Column_(pixel) and        Y_(pixel)=Y_(origin)+Y_(size)×Row_(pixel).    -   The available nadir images are checked to see if they cover the        same point on the ground as the grid pixel being filled. If so,        a mathematical formula is used to determine where that point on        the ground projects up onto the camera's pixel image map and        that resulting pixel value is then transferred to the grid        pixel. During this selection process, two important steps are        taken:    -   When selecting the image to use to provide the pixel value, a        mathematical formula is used to select an image that minimizes        building lean—the effect where buildings appear to lean away        from the camera. This is accomplished in a number of ways, but        the most common is to pick the image where the grid pixel        reprojects as close to the camera center, and hence as close to        that camera's nadir point, as possible.    -   When determining the source pixel value to use, the ground        elevation is taken into account to ensure the correct pixel        value is selected. Changes in elevation cause the apparent        location of the pixel to shift when captured by the camera. A        point on the ground that is higher up will appear farther from        the center of the image than a point on the ground in the same        location that is lower down. For instance, the top of a building        will appear farther from the center of an image than the bottom        of a building. By taking the ground elevation into account when        determining the source pixel value, the net effect is to        “flatten” the image out such that changes in pixel location due        to ground elevation are removed.

Because the rectilinear grids used for the ortho-mosaic are generallythe same grids used for creating maps, the ortho-mosaic images bear astriking similarity to maps and as such, are generally very easy to usefrom a direction and orientation standpoint. However, since they have anappearance dictated by mathematical projections instead of the normalappearance that a single camera captures and because they are capturedlooking straight down, this creates a view of the world to which we arenot accustomed. As a result, many people have difficulty determiningwhat it is they are looking at in the image. For instance, they mightsee a yellow rectangle in the image and not realize what they arelooking at is the top of a school bus. Or they might have difficultydistinguishing between two commercial properties since the only thingthey can see of the properties in the ortho-mosaic is their roof tops,whereas most of the distinguishing properties are on the sides of thebuildings. An entire profession, the photo interpreter, has arisen toaddress these difficulties as these individuals have years of trainingand experience specifically in interpreting what they are seeing innadir or ortho-mosaic imagery.

Since an oblique image, by definition, is captured at an angle, itpresents a more natural appearance because it shows the sides of objectsand structures—what we are most accustomed to seeing. In addition,because oblique images are not generally ortho-rectified, they are stillin the natural appearance that the camera captures as opposed to themathematical construction of the ortho-mosaic image. This combinationmakes it very easy for people to look at something in an oblique imageand realize what that object is. Photo interpretation skills are notrequired when working with oblique images.

Oblique images, however, present another issue. Because people havelearned navigation skills on maps, the fact that oblique images are notaligned to a map grid, like ortho-mosaic images, makes them much lessintuitive when attempting to navigate or determine direction on animage. When an ortho-mosaic is created, because it is created to arectilinear grid that is generally a map grid, the top of theortho-mosaic image is north, the right side is east, the bottom issouth, and the left side is west. This is how people are generallyaccustomed to orienting and navigating on a map. But an oblique imagecan be captured from any direction and the top of the image is generally“up and back,” meaning that vertical structures point towards the top ofthe image, but that the top of the image is also closer to the horizon.However, because the image can be captured from any direction, thehorizon can be in any direction, north, south, east, west, or any pointin between. If the image is captured such that the camera is pointingnorth, then the right side of the image is east and the left side of theimage is west. However, if the image is captured such that the camera ispointing south, then the right side of the image is west and the leftside of the image is east. This can cause confusion for someone tryingto navigate within the image.

Additionally, because the ortho-mosaic grid is generally a rectilineargrid, by mathematical definition, the four cardinal compass directionsmeet at right angles (90-degrees). But with an oblique image, because itis still in the original form the camera captured and has not beenreprojected into a mathematical model, it is not necessarily true thatthe compass directions meet at right angles within the image. Because inthe oblique perspective, you are moving towards the horizon as you moveup in the image, the image covers a wider area on the ground near thetop of the image as compared to the area on the ground covered near thebottom of the image. If you were to paint a rectangular grid on theground and capture it with an oblique image, the lines along thedirection the camera is pointing would appear to converge in thedistance and the lines across the direction of the camera is pointingwould appear to be more widely spaced in the front of the image thanthey do in the back of the image. This is the perspective view we areall used to seeing—things are smaller in the distance than close up andparallel lines, such as railroad tracks, appear to converge in thedistance. By contrast, if an ortho-mosaic image was created over thissame painted rectangular grid, it would appear as a rectangular grid inthe ortho-mosaic image since all perspective is removed as an incidentalpart of the ortho-mosaic process.

Because of these fundamental differences in perspective and appearance,the creation of an ortho-mosaic image by the process described abovedoes not work well for oblique images. Because the camera's optical axis(an imaginary line through the center of the lens or optics that followsthe aim of the camera) is typically pointed at an angle of 45-degrees ormore from nadir (pointed 45-degrees or more up from straight down), theeffects of building lean, elevation differences, and non-square pixelsare all exaggerated—effects that are considered negative qualities in anortho-mosaic image. In the ortho-mosaic industry, requirements aregenerally placed on the image capture process such that they limit theamount of obliqueness to as little as 5-degrees from nadir so as tominimize each of these negative effects.

In addition, if the admirable properties of an oblique image are to bemaintained, namely seeing the sides of structures and the naturalappearance of the images, then clearly a process that attempts to removevertical displacements, and hence the sides of the buildings, and onethat warps the image to fit a rectilinear grid is not a viable choice. Anew process is needed, one which meets the following desirable qualitiesin an effort to preserve the admirable properties of the oblique image:

-   -   If the oblique perspective is to be maintained, the pixels        cannot be aligned to a rectilinear grid, or even a trapezoidal        grid. Instead, the pixels are preferably aligned to the natural        perspective that a camera captures.    -   As part of the oblique perspective, the pixels in the image        cannot all measure the same size on the ground, as pixels in the        foreground of the image cover a much smaller area on the ground        than pixels in the background of the image—that is by definition        part of the natural perspective of a camera.    -   Because the pixels are so far from nadir, the effects of        building lean become extreme and the standard solutions employed        in the ortho-mosaic process do not do an adequate enough job        compensating for this effect—new techniques must be developed to        better compensate for this effect.    -   If the effects of changes in elevation are backed out, the        resulting image has a very unnatural appearance—the vertical        sides of buildings can warp and twist, which is something we are        not accustomed to seeing and therefore, when looking at such an        image, we have a tendency to “reject” it. Thus, to keep the        buildings, structures, and objects within an image looking        natural, it is preferable to leave the effects of elevation in        the perspective of the image and instead account for it in        another manner.

Because of these issues, the common practice in the industry is toprovide oblique imagery as a series of individual images. However, someof the same benefits of the ortho-mosaic also apply to an oblique-mosaic(an image created from a collection of overlapping or adjacent obliqueimages), namely the fact that the mosaic covers a larger geographic areathan each or any of the individual images that were used to create it.This invention details a means by which a quality oblique-mosaic can becreated, overcoming the above limitations.

SUMMARY OF THE INVENTION

This invention allows for the creation of an oblique-mosaic image thathas both a natural appearance and is preferably geo-referenced tomaintain the ability to measure and determine geographic coordinates.While the preferred embodiment applies this invention to aerial obliqueimagery, the invention will also work with non-aerial oblique imagerycaptured in a variety of ways, including but not limited to camerasmounted obliquely on a vertical pole, hand-held cameras aimed obliquely,and cameras mounted at oblique angles on an underwater probe.

In one embodiment, the present invention is directed to a method forcreating an oblique-mosaic image from a plurality of source obliqueimages of a geographical area. The source oblique images are preferablycaptured oblique images. In this method, a desired area to be imaged andcollected into an oblique-mosaic image is identified, and a mathematicalmodel of a sensor of a virtual camera is created. The virtual camera hasan elevation greater than an elevation of the area to be imaged. Themathematical model has an oblique-mosaic pixel map for the sensor of thedesired area. A surface location for each pixel included in the obliquemosaic pixel map is determined and/or assigned. Then, at least onesource oblique image pixel of the area to be imaged is reprojected foreach pixel included in the oblique-mosaic pixel map to thereby create anoblique-mosaic image of the desired geographical area. Thus, theoblique-mosaic image is a composite image formed from pixels reprojectedfrom multiple source oblique images.

In one version of the invention, multiple source oblique imagesrepresent the same surface location. In this embodiment, the method canfurther comprise steps for selecting which pixels and/or source obliqueimages to utilize in forming the oblique-mosaic image. For example, themethod can further comprise the step of selecting source oblique imagesof the surface location captured at an oblique angle and compassdirection similar to the oblique angle and compass direction of thevirtual camera. In yet another version, the pixels of each sourceoblique image that represent the same surface location are compared todetermine which source pixel is most representative of the surfacelocation, the more representative pixel to be included in theoblique-mosaic image.

In one version of the invention, the step of assigning is furtherdefined by as projecting each pixel through the perspective of thevirtual camera to determine a corresponding surface location for eachpixel in the oblique-mosaic pixel map.

In another version of the invention, the step of reprojecting is furtherdefined by reprojecting the pixel to match the size and shape of therepresented surface location as taken from the elevation, compassdirection, and oblique angle of the virtual camera.

In yet another version of the invention, the step of reprojecting isfurther defined by removing the effects of elevation from the sourceoblique images prior to reprojection and then adding the effects ofelevation to the oblique-mosaic image after reprojection.

In yet another version of the invention, the method includes the step ofminimizing lean of vertical structures within the oblique mosaic image.This can be accomplished in a variety of manners, such as creating anelevation model from the source oblique images taking into account thevertical structures, and reprojecting at least one source oblique imagepixel of the area to be imaged utilizing the elevation model.Alternatively, lean of vertical structures can be minimized by matchingvertical structures in multiple source oblique images and shiftingpixels apparent location in at least one of the source oblique images bya relative height above a ground model.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a schematic view of a real camera captured scene reprojectedto a “virtual camera”, i.e., a mathematical model that describes acamera larger than the real camera and higher in elevation than the realcamera.

FIG. 2 is an illustration of an exemplary software/function flow chartof a method for making an oblique-mosaic image using a methodologywhereby separate obliquely captured aerial images are combined into theoblique-mosaic image.

FIG. 3A is a pictorial representation of a prior art oblique-mosaicimage.

FIG. 3B is an enlarged view of the pictorial representation of the priorart oblique-mosaic image of FIG. 3 a.

FIG. 4A is a pictorial representation of an exemplary oblique-mosaicimage created in accordance with the present invention.

FIG. 4B is an enlarged view of the pictorial representation of theexemplary oblique-mosaic image created in accordance with the presentinvention of FIG. 4A.

DETAILED DESCRIPTION OF THE PRESENTLY DISCLOSED AND CLAIMED INVENTION

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not limited in its applicationto the details of construction, experiments, exemplary data, and/or thearrangement of the components set forth in the following description orillustrated in the drawings. The invention is capable of otherembodiments or of being practiced or carried out in various ways. Also,it is to be understood that the phraseology and terminology employedherein is for purpose of description and should not be regarded aslimiting.

The presently claimed and disclosed invention(s) relate tooblique-mosaic images and methods for making and using the same. Moreparticularly, the presently claimed and disclosed invention(s) use amethodology whereby separate obliquely captured aerial images arecombined into at least one single oblique-mosaic image. The at least onesingle oblique-mosaic image is visually pleasing and geographicallyaccurate.

Referring now to the Figures and in particular to FIGS. 1 and 2, shownin FIG. 1 and designated by a reference numeral 10 is a schematic viewof a method for creating an oblique mosaic image 12 (See FIGS. 4A and4B). FIG. 2 is an illustration of an exemplary software/function flowchart of a method for making the oblique-mosaic image 12 using amethodology whereby separate obliquely captured aerial images arecombined into the oblique-mosaic image 12. FIG. 4A is a pictorialrepresentation of an exemplary oblique-mosaic image 12 created inaccordance with the present invention. FIG. 4B is an enlarged view ofthe pictorial representation of the exemplary oblique-mosaic image 12created in accordance with the present invention.

In general, the method identifies a desired area 15 to be imaged andcollected into the oblique-mosaic image 12. Source images are obtainedutilizing a real camera 14 capturing a scene 16 as indicated by a block18 in FIG. 2. Then, a virtual camera 20 is created as indicated by ablock 22 in FIG. 2. The virtual camera 20 is a mathematical model thatdescribes a camera larger than the real camera 14 and higher inelevation than the real camera 14 as shown in FIG. 1. The mathematicalmodel has an oblique-mosaic pixel map for the sensor of the desiredarea. A surface location is assigned to each pixel included in theoblique pixel map, and then as indicated by a block 24, source obliqueimages are selected for the ground locations and then at least onesource oblique image pixel of the area to be imaged for each pixelincluded in the oblique-mosaic pixel map is reprojected to therebycreate the oblique-mosaic image of the desired geographical area (asindicated by a block 26). Optionally, the method further includes one ormore steps to minimize the effect of vertical structure lean asindicated by a block 28.

As described hereinabove, the use of a grid reprojection methodology tocreate oblique-mosaic images is fraught with problems and is unlikely toprovide a useable image. Therefore, in order to produce the qualityoblique-mosaic image 12, an improved process must be performed. Such animproved and unique process is described and claimed herein andpreferably uses, generally, the following considerations:

-   -   First, rather than projecting onto a rectilinear grid, the input        image pixels are projected onto the “virtual camera” 20. The        virtual camera 20 is a mathematical model that describes a very        large camera, high up in the sky. Since it is a mathematical        creation, this camera 20 is not bound by the current limitations        of camera manufacturing.    -   Second, the pixels are not then reprojected to be square or even        the same size in the output image, as is the case in the        standard ortho-rectification process used when creating an        ortho-mosaic. Instead, they are reprojected to match the size        they would project onto the ground from this virtual camera        20—i.e. the natural perspective that a camera captures.    -   Third, in order to best align the combined imagery, the effects        of changes in elevation are first backed out of the input        imagery during the projection from the original input image's        camera location to the ground, and then the effects of elevation        are reintroduced during the projection from the ground up to the        virtual camera's location. The result is a natural looking image        that properly shows the contours of the land as seen from a        natural oblique perspective.    -   Finally, the effects of building lean are desirably minimized.        There are a number of different ways in which building lean can        be minimized:        -   By “steering” the cut-lines between input images down            streets such that the cut-lines do not run through a            building, which can cause conflicting building lean to            greatly distort the appearance of a building.        -   The angle of building lean can be calculated and the images            warped in order to compensate for the building lean.        -   The buildings can be matched in adjacent images and aligned            to each other.

In practice, the methodology disclosed and claimed herein, consists ofmultiple steps and data transformations that can be accomplished by oneof ordinary skill in the art given the present specification. There area number of algorithms already known in the art that steer cut-lines forortho-mosaics and could be readily adapted for use with oblique images.In addition, follow-on work could create new algorithms specificallydesigned to deal with the complexities of oblique images.

The first step to creating the oblique-mosaic image 12 according to thepresently disclosed and claimed invention requires the selection of anarea to be imaged. Generally, the area to be imaged would be a specificgeographical location. However, other areas can also be selected to beimaged into the oblique-mosaic image 12 such as building sides, walls,landscapes, mountain sides and the like.

Once a desired area to be imaged and collected into the oblique-mosaicimage 12 has been determined, the user or operator creates the “virtualcamera” 20 i.e. a mathematical construct that is capable of covering orcapturing a portion of, or the entirety of the desired area. The virtualcamera 20 is a mathematical model of a camera with the necessary camerageometry parameters (the mathematical values that define the cameramodel, for instance the number of rows and columns of the sensor plane,size of the sensor plane in millimeters, focal length in millimeters,height above ground, yaw, pitch, and roll of the optical axis) thatenable it to preferably “capture” the desired scene. For instance, avirtual camera can be devised having a very large sensor (e.g. 20,000pixel columns and 10,000 pixel rows), a standard field of view (36 mm by24 mm sensor plane and a 60 mm focal length), and be “placed” at arelatively high altitude (e.g. 30,000 feet) looking down at an obliqueangle to the north (yaw and roll of 0 and pitch of −40 degrees). In apreferred embodiment, a sensor model from a real camera is used and theuser simply modifies the parameters such that it meets the requirementsin order to “capture” the desired area.

The second step creates the resulting oblique pixel map for the virtualcamera 20. The pixel map corresponds to the virtual camera's sensor andthus typically, but not necessarily, has the same number of rows andcolumns as the virtual camera's sensor. Then, for each pixel in thepixel map image, the projective equations for the virtual camera 20 areused to project each pixel downward and away from the virtual camera 20and onto the ground, taking elevation into account when doing so(generally through the use of a mathematical elevation model of theground surface). This results in a corresponding ground location forthat virtual camera's pixel.

Once the corresponding ground location has been found, it can be used toselect which previously captured images contain image data for thatground location. This is generally done by checking to see if the groundlocation lies within the image boundaries of a previously capturedimage.

When selecting source oblique images, e.g., input captured images, inorder to achieve a desirable output image, it is important to use sourceoblique images that were captured in the same, or nearly the same,relative orientation of the virtual camera 20, in terms of obliquedownward angle and compass direction of the optical axis. While it isgenerally not an issue to use input imagery from a camera whose model isdifferent than the virtual camera's model, if that model is radicallydifferent (for instance, a line scanner versus a full frame capturedevice), it may result in an undesirable resulting image.

While this invention discusses using captured images as input to theoblique mosaic 12, it is not actually required. It is possible to use aprojected image as input to this process or even to use another obliquemosaic as input to this process. However, since this process reprojectsthe input images, it is desirable to use non-projected input images,i.e. captured images. The reason is that reprojecting already projecteddata can often lead to artifacts, sort of like rounding errors inmathematical calculations. These artifacts can create an undesirableresulting oblique-mosaic.

It is generally desirable to create the continuous oblique-mosaic 12. Inorder to do so, there must be captured image data for the entire areabeing “captured” by the virtual camera 20. This means that if multiplecaptured images are being combined to create the oblique-mosaic 12,those input images must be adjacent or more commonly, overlapping. As aresult of this overlap, it is common for there to be multiple capturedimages covering the same area on the ground. If multiple captured imagesare available for selection, a preferred captured image is chosenaccording to the selection criteria described below.

When multiple images from real cameras 14 cover the same point on theground, a selection process can be used to determine which real cameraimage should be used as input for the creation of the virtual camera'spixel map image. This selection process can be done by assigning weights(assigned numerical values) to the following input criteria, multiplyingthose weights by the normalized criterion (a value that has been scaledbetween 0 and 1), and then selecting the image with the greatest sum ofthese weight/criterion products. While any number of criteria can beused, the following three criteria have been used in the development ofthis invention:

Selection Criterion: Distance to Optical Axis

The distance between the point on the ground being selected and thepoint where the input camera's optical axis intersects the ground. Thisvalue can be normalized by dividing the distance by the maximum distanceable to be measured in the scene.

Selection Criterion: Angular Difference to Optical Axis

The difference between the following two angles: the angle of the inputcamera's optical axis (generally measured relative to the perpendicular)and the angle of the ray being cast from the virtual camera to the pointon the ground being selected (again, generally measured relative to theperpendicular). This value can be normalized by dividing by 180-degrees.

Selection Criterion: Distance to Nearest Street Centerline

The distance between the point on the ground being selected and thenearest street centerline. The street centerlines can be obtained fromvector data files such as TIGER files or other Geographic InformationSystem files. This value can be normalized by dividing by the maximumdistance able to be measured in the scene.

Once the preferred captured image has been selected, the projectiveequations for the captured image's camera are used to project from theground up to the camera, taking the ground elevation into account whendoing so. This projection through the focal point and onto the camera'ssensor plane will find a pixel row and column corresponding to the pointon the ground. As this typically does not fall on an integer row orcolumn, bilinear interpolation (an industry standard mathematicalformula for finding a single value from the proportionate proximity tothe four surrounding pixels) is used to find the pixel value for thecorresponding point on the camera's sensor plane.

This pixel value is then used to fill the pixel in the image thatcorresponds to the virtual camera's sensor plane from which the originalray was projected outward onto the ground. This process is repeated forsome or all of the remaining pixels in the virtual camera's sensorplane, resulting in an image that covers some area or the entire area onthe ground that the virtual camera 20 can “see.” Preferably, thisprocess is repeated for all of the remaining pixels in the virtualcamera's sensor plane, resulting in a complete image that covers theentire area on the ground that the virtual camera 20 can “see.”

The resulting image and its corresponding projective equations are thenstored. The resulting image can be stored in any format, including oneof many industry standard image formats such as TIFF, JFIF, TARGA,Windows Bitmap File, PNG or any other industry standard format. For thecorresponding projective equations, the following data should, in apreferred embodiment, be stored as metadata with the resulting image,either appended to the same file or in another file readily accessible(an industry standard practice is to use the same filename but with adifferent file extension):

-   -   1. The location of the camera—generally, the location of the        camera is specified by a geographic location and altitude (or        height over ground).    -   2. The orientation of the camera—generally specified by three        angles: yaw, pitch and roll (or omega, phi, and kappa).    -   3. The size of the sensor—generally specified as an overall        sensor size in millimeters (or by the size of an individual        sensor element).    -   4. The focal length of the lens—generally specified in        millimeters.    -   5. Optionally, lens distortion information—generally specified        as a principal point offset (offset between the optical axis and        the middle of the sensor) and as radial distortion terms        (polynomial coefficients that describe the amount of distortion        that occurs radially out from the optical axis).    -   6. Optionally, a ground elevation model—generally specified as a        grid of elevation values.

As discussed above, the relative perspective of the camera causes aneffect known as “building lean.” While building lean is most commonlyapplied to, and thus discussed as, buildings, it also applies to anyvertical structure in the object, such as electrical towers, trees,cars, phone booths, mailboxes, street signs, and so on. Building lean isan effect that makes it appear as if buildings that are not along theoptical axis of the camera “lean” away from the optical axis—the fartheraway from the optical axis, the greater the lean. This lean is theresult of perspective, which causes objects that are raised off theground to appear farther “back” into the image, away from the camera.Thus, the top of a building appears farther back than the bottom of abuilding. When this lean corresponds to the camera's perspective, itlooks normal. However, as part of the oblique-mosaic process, capturedimages, each with their own perspective, are combined into a singlevirtual camera's perspective.

This combination of different perspectives becomes problematic,especially when two different captured images from different vantagepoints contribute pixel values to adjacent areas in the virtual camera'spixel map image. For example: If a camera to the left of a buildingprovides the left side of a building, then that portion of the buildingwill appear to be leaning to the right (away from the camera). If acamera that is located to the right of the building provides the rightside of the building, then that portion of the building will appear tobe leaning to the left. Since the two halves of the building “lean into”each other, the resulting combined image has a building that has atriangular, rather than a rectangular, appearance.

Because building lean only affects surfaces above the surface of theground, it is generally fairly difficult to account for or correct theseeffects because in order to do so, the user must have knowledge of thepresence of the building or structure. Features that are on the grounddo not experience this building lean because the change in relativeperspective is backed out when ground elevation is taken into account inthe projective equations. The mathematical model used to define theground surface during the projective process ensures the correct groundlocation is selected. However, for objects that rise above the groundsurface, and are not represented in the mathematical model used todefine the ground surface, this relative perspective change causes thevirtual camera to “see” the building top in the wrong place—i.e. too farback from the camera.

A method for minimizing the effects of building lean, as contemplatedherein, is to transition between one input camera image and the nextinput camera image over an area where there are no structures above orbelow the ground elevation model. In one embodiment, this isaccomplished by placing the transition down the middle of a street.Thus, by having a properly weighted selection criterion for distance tostreet centerline, if there is a street in the area where two capturedimages overlap, then the transition from contributing pixels from theone captured image to contributing pixels from the second captured imagewill occur along this street, thus minimizing the effects of buildinglean.

A method for removing building lean entirely, as contemplated herein, isto provide an accurate ground elevation model taking into accountbuildings and other vertical structures. Thus, every pixel thatcomprises the image of the building is represented in the mathematicalmodel of the ground elevation model and therefore the change in relativeperspective is accounted for in the projective process. However, forthis to work well, the elevation model must be highly correlated to theinput imagery. If there is any shift in location between the imagery andthe elevation model, then the buildings will not be projected properlywhen creating the oblique-mosaic image 12.

To overcome this limitation, the preferred methodology is to create theelevation model from the imagery itself. This can be done by using anindustry standard process known as aero-triangulation, which finds theelevation of a point on the ground by comparing its location in twooverlapping captured images and using the projective equations of theircorresponding cameras to triangulate its location and elevation.Repeating this process over the entire overlap area can produce anaccurate mathematical model of the surface of not only the ground, butalso of the surface of structures and objects within the images. Moreimportantly, because this model is derived from the imagery itself, itis by definition, highly correlated to the input image.

Another method for removing building lean, contemplated for use herein,is to attempt to identify vertical structures by using an edge matchingprocess between the oblique and the corresponding nadir imagery.Vertical structures do not appear in a truly nadir image, and theybarely appear in a slightly off-nadir image. Thus, when comparing anoblique image with its corresponding nadir image, the primary differencebetween the structures that appear in the two images will be thevertical structures. By using one or more edge detection algorithms(such as an industry standard Laplacian Filter), it is possible toidentify the various structures within the two images and then isolatethe vertical edges in the oblique image by subtracting out thenon-vertical structures that also appear in the nadir image. Once thesevertical structures have been found, the pixels for those verticalstructures can be shifted to remove the effects of the change inrelative perspective. By shifting the pixel's apparent location in thecaptured oblique image by the relative height above the ground modelfound through the measuring of the vertical edges, its proper groundlocation can be determined, thus negating the effects of building lean.

It should be understood that the processes described above can beperformed with the aid of a computer system running image processingsoftware adapted to perform the functions described above, and theresulting images and data are stored on one or more computer readablemediums. Examples of a computer readable medium include an opticalstorage device, a magnetic storage device, an electronic storage deviceor the like. The term “Computer System” as used herein means a system orsystems that are able to embody and/or execute the logic of theprocesses described herein. The logic embodied in the form of softwareinstructions or firmware may be executed on any appropriate hardwarewhich may be a dedicated system or systems, or a general purposecomputer system, or distributed processing computer system, all of whichare well understood in the art, and a detailed description of how tomake or use such computers is not deemed necessary herein. When thecomputer system is used to execute the logic of the processes describedherein, such computer(s) and/or execution can be conducted at a samegeographic location or multiple different geographic locations.Furthermore, the execution of the logic can be conducted continuously orat multiple discrete times. Further, such logic can be performed aboutsimultaneously with the capture of the images, or thereafter orcombinations thereof.

Although the foregoing invention has been described in some detail byway of illustration and example for purposes of clarity ofunderstanding, it will be obvious to those skilled in the art thatcertain changes and modifications may be practiced without departingfrom the spirit and scope thereof, as described in this specificationand as defined in the appended claims below.

1. A system for creating one or more oblique-mosaic image, comprising: acomputer system running image processing software that when executed bythe computer system causes the computer system to: create a mathematicalmodel of a sensor of a virtual camera, where the virtual camera has anelevation greater than an elevation of a desired geographical area to beimaged, the mathematical model having an oblique-mosaic pixel map forthe sensor of the desired area; determine surface locations for pixelsincluded in the oblique-mosaic pixel map, wherein the surface locationsfor the pixels include first surface locations of the ground and secondsurface locations of objects above the ground; for pixels included inthe oblique-mosaic pixel map, select source oblique images of thesurface locations of the pixels captured at an oblique angle and compassdirection similar to an oblique angle and compass direction of thevirtual camera; and reproject source oblique image pixels of the area tobe imaged for pixels included in the oblique-mosaic pixel map such thatreprojected pixels have differing sizes determined by matchingprojections from the virtual camera so as to present an obliqueappearance, and thereby create the oblique-mosaic image of the desiredgeographical area.
 2. The system of claim 1, wherein determining surfacelocations for pixels included in the oblique-mosaic pixel map furthercomprises projecting the pixels in the oblique mosaic pixel map througha perspective of the virtual camera to determine a corresponding surfacelocation for the pixels in the oblique-mosaic pixel map.
 3. The systemof claim 1, wherein selecting source oblique images comprises selectingfrom multiple ones of the source oblique images representing a samesurface location.
 4. The system of claim 3, wherein the pixels of eachsource oblique image that represent the same surface location arecompared to determine which source pixel is most representative of thesame surface location, the more representative pixel to be included inthe oblique-mosaic image.
 5. The system of claim 1, wherein reprojectingsource oblique image pixels further comprises reprojecting the sourceoblique image pixels to match a size and shape of a represented surfacelocation as taken from the elevation, compass direction, and obliqueangle of the virtual camera.
 6. The system of claim 1, whereinreprojecting source oblique image pixels further comprises removingeffects of elevation from the source oblique images prior toreprojection and then adding the effects of elevation to theoblique-mosaic image after reprojection.
 7. The system of claim 1, theimage processing software that when executed by the computer systemcauses the computer system to minimize lean of vertical structureswithin the oblique-mosaic image.
 8. The system of claim 7, whereinminimizing lean of vertical structures within the oblique-mosaic imagefurther comprises creating an elevation model from the source obliqueimages taking into account the vertical structures, and reprojecting atleast one source oblique image pixel of the area to be imaged.
 9. Thesystem of claim 7, wherein minimizing lean of vertical structures withinthe oblique-mosaic image further comprises matching vertical structuresin multiple ones of the source oblique images and shifting pixelsapparent location in at least one of the source oblique images by arelative height above a ground model.
 10. The system of claim 1, whereinmetadata is stored with the oblique-mosaic image.
 11. The system ofclaim 1, wherein the oblique-mosaic image is geo-referenced.
 12. Thesystem of claim 1, wherein selecting source oblique images comprisesselecting from multiple ones of the source oblique images representing asame surface location and having meta-data indicative of a location of acamera used to capture the source oblique images and orientation of thecamera.
 13. The system of claim 1, wherein the oblique-mosaic image isan aerial oblique-mosaic image.
 14. A method for creating anoblique-mosaic image from a plurality of source oblique images,comprising the steps of: creating a mathematical model of a sensor of avirtual camera, where the virtual camera has an elevation greater thanan elevation of a desired geographical area to be imaged, themathematical model having an oblique-mosaic pixel map for the sensor ofthe desired area; determining surface locations for pixels included inthe oblique-mosaic pixel map, wherein the surface locations for thepixels include first surface locations of the ground and second surfacelocations of objects above the ground; for pixels included in theoblique-mosaic pixel map, selecting source oblique images of the surfacelocations of the pixels captured at an oblique angle and compassdirection similar to an oblique angle and compass direction of thevirtual camera; and reprojecting source oblique image pixels of the areato be imaged for pixels included in the oblique-mosaic pixel map suchthat reprojected pixels have differing sizes determined by matchingprojections from the virtual camera so as to present an obliqueappearance, and thereby create the oblique-mosaic image of the desiredgeographical area.
 15. The method of claim 14, wherein the step ofdetermining is further defined by projecting the pixels in the obliquemosaic pixel map through a perspective of the virtual camera todetermine a corresponding surface location for the pixels in theoblique-mosaic pixel map.
 16. The method of claim 14, wherein the stepof selecting is selecting from multiple source oblique imagesrepresenting a same surface location.
 17. The method of claim 16,wherein the pixels of each source oblique image that represent the samesurface location are compared to determine which source pixel is mostrepresentative of the same surface location, the more representativepixel to be included in the oblique-mosaic image.
 18. The method ofclaim 14, wherein the step of reprojecting is further defined byreprojecting the source oblique image pixels to match a size and shapeof a represented surface location as taken from the elevation, compassdirection, and oblique angle of the virtual camera.
 19. The method ofclaim 14, wherein the step of reprojecting is further defined byremoving effects of elevation from the source oblique images prior toreprojection and then adding the effects of elevation to theoblique-mosaic image after reprojection.
 20. The method of claim 14,further comprising the step of minimizing lean of vertical structureswithin the oblique-mosaic image.
 21. The method of claim 20, whereindetermining surface locations for pixels included in the oblique-mosaicpixel map, wherein the surface locations for the pixels include firstsurface locations of the ground and second surface locations of objectsabove the ground, comprises: creating a mathematical model of firstsurface locations of the ground and of the second surface locations ofobjects above the ground in the geographical area, based on two or moreseparately captured oblique images having an overlapping image areadepicting the ground and the objects above the ground in thegeographical area, by using aero-triangulation, comprising: for aplurality of points in the overlapping image area: comparing a firstlocation of the point in a first one of the source oblique images with asecond location of the point in a second one of the source obliqueimages; and triangulating real-world location and elevation of the pointusing information indicative of one or more camera that captured thesource oblique images and the comparison of the first location and thesecond location.
 22. A system for creating one or more oblique-mosaicimage, comprising: a computer system running image processing softwarethat when executed by the computer system causes the computer system to:create a mathematical model of surfaces of the ground and of two or morestructures in a geographical area, based on two or more separatelycaptured oblique images having an overlapping image area having pixelsdepicting the ground and the two or more structures in the geographicalarea, by using aero-triangulation, comprising: for a plurality of pointsin the overlapping image area: comparing a first location of the pointin a first one of the two or more oblique images with a second locationof the point in a second one of the two or more oblique images; andtriangulating real-world location and elevation of the point usinginformation indicative of one or more camera that captured the two ormore oblique images and the comparison of the first location and thesecond location; and reproject one or more pixels from the two or moreseparately captured oblique images onto the mathematical model to createan oblique-mosaic image of the geographical area.
 23. The system ofclaim 22, wherein the reprojected pixels have differing sizes so as topresent an oblique appearance.
 24. The system of claim 22, wherein thecomputer system runs image processing software that when executed by thecomputer system causes the computer system to receive a selection of thegeographical area to be modeled.
 25. The system of claim 22, wherein thecomputer system runs image processing software that when executed by thecomputer system causes the computer system to obtain the two or moreseparately captured oblique images.
 26. The system of claim 22, whereinthe computer system runs image processing software that when executed bythe computer system causes the computer system to utilize the createdmathematical model to eliminate the appearance of building lean in anoblique mosaic image.
 27. The system of claim 22, wherein one or more ofthe two or more separately captured oblique images are aerial obliqueimages.